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Abstract 

In this paper we propose a method for the adaptive complex assessment (ACA) of the higher-order thinking (HOT) 
skills needed by students for problem solving, and we examine the impact of the method on the development of HOT 
skills in a problem- based learning (PBL) environment. Complexity in the assessment is provided by initial, 
formative, and adaptive assessments of HOT skills; assessments of collaborative skills; and summative assessments 
of HOT skills. Adaptability in the assessment is provided through the dynamics of an instructor’s assessments based 
on developing HOT skills; through a flexible choice of control tests and instructional problems for students and 
collaborative groups; and through the self-formation of heterogeneous, collaborative HOT skill groups. The 
assessment fosters the development of HOT and collaborative skills through a combination of personalized and 
collaborative problem-based learning (PBL). The three-stage assessment process guides the development of HOT 
skills during subject study through PBL. The focus of the first stage is on developing the HOT skills of students 
through personalized PBL. The second stage is devoted to developing HOT skills and collaborative skills through 
collaborative PBL. The third stage is devoted to assessing collaborative skills and constructing summative 
assessments of students. The proposed calculations for the coefficients of HOT skill development serve as a 
constructive means of exploring the impact of the ACA method on the HOT skill development of students. 

Keywords: adaptive complex assessment; higher-order thinking skills 

1. Introduction 

Among the various learning strategies introduced in science education Problem- based learning (PBL) is an 
important student-centered strategy. It is directed towards the development of the higher-order thinking (HOT) skills 
that students need for problem solving. For this study, we consider two layers of HOT skills: analytical thinking 
skills and creative thinking skills. Analytical thinking skills consist of ordering, comparing, contrasting, evaluating, 
and selecting. Creative thinking skills involve problem finding (identifying the problem), efficiency (producing many 
ideas), flexibility (producing a broad range of ideas), originality (producing uncommon ideas), and elaboration 
(developing ideas) (Hmelo-Silver, 2004; Bednarz, 2011; Cottrell, 2011, 2013). Obtaining information about the 
development of the knowledge, and skills, of a student or group of students is provided by assessment, which serves 
as an important educational tool (Brown, 2004; Hargreaves, 2007). There are different methods for assessing 
learning based on home work, examinations, and assignments (Ramsden, 2000; Hamey et. al., 2015). However, these 
methods do not provide for customization when assessing student performance in a PBL environment. In other words, 
adaptive assessment is not realized. As a result, students are not induced engage in PBL. 

Motivation for students to develop their HOT skills should be provided in PBL environment. It can be attained 
through specific forms of adaptive assessment. A method for the adaptive complex assessment (ACA) of the HOT 
skills of students is proposed in this paper, which also examines the impact of the assessment on the development of 
HOT skills in a problem-based collaborative learning environment. The paper develops an approach to the 
assessment ofHOT skills that can be acquired by students during adaptive PBL (Raiyn &Tilchin, 2015). 
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2. Related Research 

The goals of teaching and learning with the PBL model are to develop HOT skills, promote the adoption of a 
problem solving process, and foster collaborative skills in students. The assessment of PBL can serve as a 
constructive mean of attaining these goals. Various aspects of an assessment process are reviewed here. According to 
Knight (2000) a systematic approach to the assessment of student learning is needed. Ma and Zhou (2000) suggested 
an approach to the assessment of learning outcomes aimed at encouraging students to participate in the whole 
learning process. Moallem (2007) distinguished the following stages of the assessment process: the initial assessment, 
progress assessment, and product assessment. Tillema (2010) confirmed that the focus of formative assessment is to 
promote learning. Papanastasiou (2014) defined the adaptive assessment of a student as a type of assessment that is 
customized to his or her previous performance. Savery (2006) affirmed that one of the objectives of PBL is the 
adoption of a problem-solving process. Macdonald and Savin-Baden (2004) marked the need for specific assessment 
methods for PBL. Lovie-Kitchin (2001) claimed that assessment should be coordinated with the PBL process. Ma 
(2002) developed a decision support system for assessing PBL. Debbie (2009) emphasized the influence of 
individual assessments on group composition. Swan, Shen and Hiltz (2006) stressed the role of assessment in 
encouraging of collaborative learning. McLoughlin and Luca (2002) suggested a learner-centered approach to 
developing collaborative skills through assessment. Analysis survey of the literature shows that there is no method in 
existence for the adaptive complex assessment of students in a problem-based collaborative learning environment. 
The proposed method will have a significant impact on the development of HOT skills and the collaborative skills of 
students, due to the three -staged assessment process for PBL 

• the complexity of the assessment 

• the self-formed, heterogeneous, collaborative, HOT skills groups of students 

• the customization of assessments for the performance of students in a PBL environment. 

3. An Adaptive Complex Assessment Method for HOT Skill Development 

The goal of the ACA method is to promote the guided development of students’ HOT skills through a three-stage 
assessment process for PBL. The first stage involves conducting the initial assessments of HOT skills for individual 
students, arriving at formative assessments for students’ HOT skills, and arranging for self-forming heterogeneous, 
collaborative HOT skills groups of students on the basis of the formative assessments and the instructor requirements 
for forming the collaborative groups. The second stage involves conducting adaptive assessments of HOT skills for 
students and the collaborative groups, and assessing HOT skills of individual students and collaborative groups. The 
third stage consists of assessing collaborative skills and conducting summative assessments of students. The 
adaptability of assessment is expressed through the customization of the assessment of student performance in a 
problem-based collaborative learning environment. The complexity of the assessment follows from conducting initial 
and formative assessments of the HOT skills of individual students, and the collaborative groups, assessments of 
collaborative skills, and summative assessments of HOT skills. Assessment is realized through students’ replies to 
control questions and their solving of instructional problems prepared by an instructor. 

3.1 The First Stage of the PBL Assessment Process 

3.1.1 Conducting Initial Assessments of HOT Skills for Individual Students 

The initial assessments of HOT skills are made during this stage of the PBL assessment. They express the 
preferences of instructors for developing certain skills in students. Analytical skills are preferred over creative skills 
at this stage since they provide the basis for the subsequent development of creative skills. The skill termed 
“evaluating” from among the analytical types of thinking skills shows the greatest need for development, since it is a 
form of critical thinking. Consequently, the maximal initial assessment is conducted for this skill. Initial assessments 
of creative skills are schedule order of preference for the development of the problem- solving abilities of students. 

Example 1: The initial assessments are presented in Table 1. 
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Table 1. The Initial Assessments of HOT Skills for Individual Students 


The type of thinking skills 

Skills name 

The fixed assessments 


Evaluating 

24 


Selecting 

18 


Comparing 

13 

Analytical (70%) 

Contrasting 

10 


Ordering 

5 


Efficiency 

10 


Flexibility 

8 

Creative (30%) 

Originality 

5 


Problem finding 

4 


Elaboration 

3 


3.1.2 Determination of the Formative Assessments of Students’ HOT Skills 

Formative assessments make up the first level of the HOT skills development of the study group students. These 
assessments serve as the basis for setting up the adaptive forming heterogeneous collaborative HOT skills groups of 
students and for conducting the adaptive assessments of students’ HOT skills. 

Example 2: The determined formative assessments are represented by Table 2. 

Table 2. The Formative Assessments of Student’s HOT Skills 



The questions and 
instructional problems 

Skill name 


SI 

S2 

S3 

S4 

S5 

S6 


qn 

Evaluating 

12% 

8 

10 

4 

7 

5 

8 


0 l 2 

Evaluating 

12% 

9 

9 

5 

6 

7 

9 


<421 

Selecting 

9% 

5 

7 

7 

8 

5 

8 

Analytical 

<422 

Selecting 

9% 

6 

5 

5 

7 

4 

7 

skills 

031 

Comparing 

13% 

9 

8 

6 

11 

7 

8 


qu 

Contrasting 

10% 

7 

9 

4 

7 

8 

4 


qsi 

Ordering 

5% 

3 

2 

1 

3 

4 

1 



The formative assessments of 

47 

50 

33 

49 

40 

45 



analytical skills 









Pi 

Efficiency 

10% 

6 

9 

5 

9 

6 

7 


P 2 

Flexibility 

8% 

7 

6 

4 

7 

5 

5 

Creative 

P 3 

Originality 

5% 

4 

4 

2 

3 

3 

3 

skills 

P 4 

Problem finding 

4% 

2 

2 

0 

1 

0 

2 


P 5 

Elaboration 

3% 

1 

2 

1 

1 

2 

1 



The formative assessments of 

20 

23 

12 

21 

16 

18 



creative skills 










The formative assessments of 

67 

73 

45 

70 

56 

63 



HOT skills 









3.1.3 The Self- Formation of Heterogeneous Collaborative, HOT Skills Groups of Students 

Each student chooses suitable partner for collaborative PBL by taking into account the formative assessments of 
HOT skills. The requirements for forming the collaborative groups are determined by the instructor. The 
requirements are: 

• maximum mutual supplementation of the HOT skills of students in each a collaborative group 

• taking into account the individual characteristics of students 

• keeping to the allowable maximum number of students in a group. 

These requirements facilitate the development of skills in cooperating students through interactions that compensate 
for the lack of personal HOT skills. As a result of mutual choices, the self-formation of a collaborative group is 
realized. 
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Example 3: According to the requirements for forming the collaborative groups and according to the formative 
assessments of the HOT skills of students (Table 2), two collaborative groups formed themselves. The first 
collaborative group gj includes students S|. s 3 , and s 5 . The second collaborative group g 2 includes students s 2 , s 4 , and 

s 6 . 

3.2 The Second Stage of the PBL Assessment Process 

3.2.1 Conducting Adaptive Assessments of HOT Skills for Students and Collaborative Groups 

Selection adaptive instructional problems for collaborative groups of students is done by taking into account the level 
of development of their HOT skills: a group with a higher level of skill development should receive more complex 
instructional problems to solve. Hence, more highly adaptive assessments of creative skills should be given to groups 
that receive more complex instructional problems. 

Example 4: On the basis of the data from Table 2, the formative assessments of the analytical skills of the first and 
second collaborative groups are 120, and 144, respectively. The formative assessments of the creative skills of the 
first and second collaborative groups are 48, and 62, respectively. The total formative assessments of the HOT skills 
of the first and second collaborative groups are 168, and 206, respectively. Consequently, the second collaborative 
group should receive more complex instructional problems, and more highly adaptive assessments of creative skills 
for students in this group should be conducted. The adaptive assessments of HOT skills for corresponding 
collaborative groups are presented in Table3. 

Table 3. Adaptive Assessments of HOT Skills for Collaborative Groups 


The type of thinking skill 

Assessments for the first 

Assessments for the 


group 

second group 

Analytical 

30 

20 

Creative 

50 

60 


Adaptive assessments based on formative assessments stimulate and facilitate the acquisition of HOT skills. The 
adaptive assessments of HOT skills for individual students in the first and second collaborative groups are presented 
in Table 4. 

Table 4. Adaptive Assessments of HOT Skills for Individual Students 


The type of 
thinking skills 

Skill name 

The adaptive assessments for individual 
students 

The first group The second group 


Evaluating 

3 

2 


Selecting 

7 

3 


Comparing 

5 

4 

Analytical 

Contrasting 

6 

5 


Ordering 

9 

6 


Originality 

8 

9 

Creative 

Efficiency 

12 

12 


Flexibility 

14 

7 


Problem finding 

7 

14 


Elaboration 

9 

18 


A fixed assessment of collaborative skills is introduced to stimulate interactions among students during collaborative 
problem solving. This assessment is set equal to 20%. 

3.2.2 Assessment of the HOT Skills of Individual Students 

Assessments received by students as a result of PBCL are presented by Table5. 


Published by Sciedu Press 


15 


ISSN 1925-0746 E-ISSN1925-0754 




http://wje.sciedupress.com 


World Journal of Education 


Vol. 6, No. 2; 2016 


Table 5. Assessments of HOT Skills of Individual Students 



The questions and Skill names 

instructional 

problems 

The first group 

SI S3 S5 

The second 

S2 S4 

group 

S 6 


ql 

Evaluating 

3 

2 

2 

2 

1 

2 


q 2 

Selecting 

6 

5 

4 

3 

3 

2 


q3 

Comparing 

5 

3 

5 

4 

3 

3 

Analytical 

q4 

Contrasting 

5 

6 

5 

4 

5 

4 

skills 

q5 

Ordering 

8 

6 

8 

6 

5 

4 



Assessment of analytical skills 

27 

22 

24 

19 

17 

15 


pi 

Originality 

7 

5 

6 

9 

9 

8 


P2 

Efficiency 

12 

9 

10 

11 

10 

10 

Creative 

P3 

Flexibility 

13 

12 

12 

6 

7 

6 

skills 

P4 

Problem finding 

7 

4 

5 

13 

12 

11 


P5 

Elaboration 

8 

6 

9 

17 

16 

15 



Assessment of creative skills 

47 

36 

42 

56 

54 

50 



Assessment of HOT skills 

74 

58 

66 

75 

71 

65 


3.3 The Third Stage of the PBL Assessment Process 

3.3.1 Assessment of Collaborative Skills 

The collaborative groups are skill heterogeneous within the requirement for the maximal mutual supplementation of 
skills within a group, which fosters and facilitates skill-sharing among students. The assessment of collaborative 
skills is based on an analysis of the assessments of the analytical and creative HOT skills of students after 
collaborative problem solving. The objective of the analysis is to identify the student (or students) who scored 
highest for HOT skills on this assessment, which would be the result of the acquisition of HOT skills due to 
collaborative problem solving. This can serve as a measure of the success of collaboration for students, with peers in 
a group or with members of other collaborative groups. A student with a maximal assessment of HOT skills 
presumed to possess the best collaborative skills. The assessments of the collaborative skills of students in a group 
are calculated proportionally to the assessments of HOT skills based on the fixed assessment of collaborative skills. 
Example5: The fixed assessment of collaborative skills is 20%. This assessment is determined by student S 2 , since the 
assessment of the HOT skills of this student is maximal at 75 (Table 5). Hence, the assessment of the collaborative 
skills achieved by student S 2 equals 20%. Student S 3 has the worst collaborative skills, since the assessment of HOT 
skills of this student is minimal at 58(Table 6 ). The collaborative skills of student S 3 equal 15%. The assessments of 
the collaborative skills of the remaining students are calculated analogously. The results of the calculations are 
shown in the third row of Table 7. 

3.3.2 Conducting of Summative Assessments of Students 

The combined summative assessments characterize the outcomes of PBL. These assessments are determined by a 
summation of the first stage assessments of HOT skills (Table2), the second stage assessments of HOT skills 
(Table5), and the assessments of collaborative skills (Example5). The results from calculating the combined 
summative assessments of students are presented in the penultimate row of Table 6 . The final summative 
assessments are presented in the last row of Table 6 . 


Table 6. The Combined Summative Assessment 


The assessment type 

The first group 
SI S3 

S5 

The second 

S2 S4 

group 

S 6 

The first stage assessments of HOT skills 

67 

45 

56 

73 

70 

63 

The second stage assessments of HOT skills 

74 

58 

66 

75 

71 

65 

Assessments of collaborative skills 

19 

15 

17 

20 

18 

16 

The combined summative assessments 

160 

118 

139 

168 

159 

144 

The final summative assessments 

80 

59 

70 

84 

79 

72 
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3.4 Impact of the ACA Method on HOT Skills Development 

The impact of the ACA method on student’s HOT skill development is provided by the sequential application of a 
three staged assessment process, which consists of (1) assessing the development of students’ HOT skills based on 
the results of the first assessment, (2) assessing the development of students’ HOT skills based on the result of the 
second assessment, and (3) assessing the development of the HOT skills of the collaborative groups. The results of 
the initial assessment make it possible to assess the development of students’ HOT skills by comparing them (Table 
1) with the results of formative assessment (Table 2) using the formula: 

5(k i ) = (g f (k i )-g(k,))/g(k i ) , - 1 < 5(k ; ) < 0 ( 1 ) 

where 

S(kj) is the coefficient of the development of skill k; in a student; 
g*(ki) is the formative assessment of k, skill developed by a student; 
g (k;) is the initial assessment of skill k;. 

Example 6 : The formative assessments of the “evaluating” skill for students Si, S 3 , and S 5 are 17, 9, and 12, 
respectively (Table 2). The initial assessment of this skill was equal to24 (Table 1). Therefore, the values of the 
coefficients for the development of the “evaluating” in these students, according to formula (1) are -029, -063, and 
-0.50, respectively. The values of the coefficients for the development of other skills are determined analogously. 

The results of the second stage of assessment make it possible to assess students’ developing HOT skills by 
comparing the results of the individual student assessments (Table5) with the result of the adaptive assessments of 
individual students (Table4) using the formula (1). 

Example 7: The assessment scores for the “evaluating” for students s 1; s 3 , and s 5 are, 3, 2, and 2, respectively (Table 
5). The adaptive assessment of this skill is 3 (Table 4). Therefore, the values of the coefficients for the development 
of “evaluating” in these students, according to formula (1), are 0, -0.33, and -0.33, respectively. The values of the 
coefficients for development of other skills are determined analogously. A comparison of the values of the 
coefficients for the development of “evaluating” calculated relative to the first assessment stage (Example 6 ) and the 
second assessment stage (Example 7), leads to the conclusion that the ACA method develops this skill in the students 
of the first collaborative group. Assessing the development of the analytical and creative skills associated with 
collaborative PBL is realized by comparing corresponding values of the coefficients for the development of these 
skills from the first and second assessment stages. A coefficient for development 5(k a ) of an aggregate of analytical 
or creative skills in a student is determined by using the formula: 

S(k) = (g(k a ) - g b (k a )) / g b (k i ) , - 1 < 5(k a ) < 0 ( 2 ) 

where 

g(k a ) is the assessment of an aggregate of analytical or creative skills possessed by a student and, 
g b (k a ) is the basic (initial or adaptive) assessment of the skill aggregate. 

Example 8: The formative assessments of the analytical skills possessed by students Si, s 3 , and S5 owing to realizing 
the first assessment stage are 47, 33, and 40, respectively (Table 2). The initial assessment of these skills was equal 
to70 (Table 1). Therefore, the values of the coefficients for the development of analytical skills in these students, 
according to formula (2) are -0.32, -0.53, and -0.43 respectively. The values of the coefficients of the development of 
creative skills are calculated analogously. 

An analysis of the data from Table 7 leads to the conclusion that the ACA method develops analytical and creative 
skills in students. 
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Table 7. The Coefficient Values for the Development of Aggregates of Skills in Students 


The first The second 

Group_group 


The name of assessment stage 

The name of development 
coefficient 

SI 

S3 

S5 

S2 

S4 S6 

The first assessment stage 

Coefficient of analytical 
skills development 

-0.32 

-0.53 

-0.43 

-0.29 

-0.30 -0.36 


Coefficient of creative 
skills development 

-0.33 

-0.60 

-0.47 

-0.23 

-0.30 -0.4 

The second assessment stage 

Coefficient of analytical 
skills development 

-0.10 

-0.27 

-0.20 

-0.05 

-0.15 -0.25 


Coefficient of creative 
skills development 

-0.06 

-0.28 

-0.16 

-0.07 

-0.10 -0.17 


4. Conclusion and Future Work 

A method for adaptive complex assessment guiding the development of students’ HOT skills through a three-staged 
assessment process of problem-based collaborative learning is proposed. The adaptability of the assessment is 
achieved by enabling the customization of the assessments and allowing for a flexible choice of control tests and 
instructional problems for students and collaborative groups. Complexity in the assessment is achieved by: initial and 
formative assessments of the HOT skills of individual students, adaptive assessments of HOT skills in individual 
students and collaborative groups, assessments of collaborative skills, and summative assessments of HOT skills. 
The method promotes: the adoption of a problem- solving process by students; the self-formation of productive, 
heterogeneous, collaborative HOT skill groups, and the effective development of HOT and collaborative skills in 
students. The impact of the adaptive complex assessment on PBL is determined by calculating coefficients for HOT 
skill development in students. Further research will be directed towards developing mechanism for the adaptive 
self-formation of collaborative groups, and towards the creation of a computer-supported tool for managing HOT 
skill development in students. 
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